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ABSTRACT 


~ 


A recent survey of the Navy Personnel Research Community 
is subjected to a comparative analysis of the various program 
elements in the research program in support of General 
Operational es 43, Personnel Logistics, for military 
Significance and investment risk. The qualitative appraisal 
of the individuals polled are quantified, using the Ford 
procedure, and the resultant data analyzed, using a multiple 
regression model, to determine how much agreement/disagree- 
ment there is in the community, where any difference might be, 
and from which program elements the differences stem. The 
results should provide assistance for the decision-maker in 
determining the right project areas for the Navy Personnel 


Research Program to be maximally effective. 
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I. INTRODUCTION 


A previous study [Arima, 4] of the hats Personnel 
Research Laboratories noted that the "evaluation of the 
research and technology end of the R&D spectrum is the 
difficult problem, that this has to be done subjectively/ 
qualitatively, and that some sort of panel of experts is an 


appropriate mechanism." 


This is because generally appli- 
cable, hard measures of the effectiveness of research 
programs do not exist; thus, the eventual determination is 
based on the judgmental appraisals of those making the 
evaluations. 

One common “panel of experts" is an advisory board; 
another, which stems from the work of Pelz and Andrews [6], 
is the evaluation of effectiveness and quality by peer 
ratings. This latter method was investigated and reported 
by Arima and Mister [5]; “the purpose of this study was to 
develop, validate and test apply a procedure for obtaining 
qualitative judgments economically and efficiently...and 


concluded that the Ford procedure could be used to obtain 


scaled qualitative judgments..with accuracy, efficiency and 


1 Naval Postgraduate School Report NPS55AA71071A, 
Evaluation and Innovation in the Navy's Personnel Research 
Laboratories: I. Preliminary Consideration, James K. Arima, 
July J97Ly De 30e 





economy. "2 It suggested that the Ford procedure could be used 


in analyses with “quality of life" variables for which con- 
ventional measurements do not exise, and specifically for 
analyses "based on the relationship between funding and user 
ratings, professional estimates of feasibility of finding 
solutions through research, and the resources actually being 


programmed for the projects."? 


This initiated a present 
analysis of the Navy's Personnel Research Program of which 
this thesis is a part. 

The objectives of this analysis were to use the Ford 
procedure and multiple regression to determine (1) how much 
agreement/disagreement there is in the Navy personnel 
research community, (2) where any difference might be, and 
(3) from which program elements the differences stem. The 
underlying premise was that the right project areas must be 
under research in order for the Navy Personnel Research 
Program to be maximally effective. Determination of the 
right project areas is beyond the scope of this report, but 
the information contained herein and other feedback from the 


Navy Personnel R&D community should provide assistance for 


the decision-maker in determining the right project ageas. 


2. Naval Postgraduate School Report NPS55AA72051A, 
Evaluation and Innovation in the Navy's Personnel Research 
Laboratories: II. Development, Validation and Trial 
Application of a Computer Program to Facilitate Judqmental 
Appraisal, James K. Arima and Richard W. Mister, May 1972. 


3 Ibid, p. 30. 





II. METHOD 


A. DATA COLLECTION 

The first step in the analysis was to collect the 
opinions of the Navy RDT&E community on the various program 
elements in the research program in support of General Oper- 
ational Requirement (GOR) 43, Personnel Logistics. In order 
to do this, using the — rating method as suggested, 263 
members of the nine panels listed in Table I which represent 
a cross-section of the Navy Personnel Research community were 


chosen as the panels of experts. 


TABLE I 


Navy Research Personnel Community 


Panel Title 
100 Managers, Pers A3 (Personnel Research) 
200 Managers, ileadquarters Department of Navy 
300 Managers, NAVPERSLAB, WASH. D.C. 
400 Professionals, NAVPERSLAB, SDGO 
500 Professionals, NAVPERSLAB, WASH. D.C. 
600 Managers, NAVPERSLAB, SDGO 
700 Users (U.S. Naval Officers) 
800 Members, NAVPERSLAB Advisory Board 
900 Civilian Contractors (Personnel Research) 


Questionnaires (see Appendix A) were sent to each 
member requesting they independently rank the 18 typical 
research program elements listed in Table II for their 
military significance, which refers to the needs of the Navy, 
and investment risk, which refers to the probability of a 


successful research effort independent of its military 


significance. 





TABLE II 


Navy Personnel R&D Program Elements 


1. Drug Abuse 
2. Education and Training 
3. Occupational R&D 
4. Manpower/Personnel Cost Systems 
5. Manpower Requirements 
6. Personnel Management Systems 
7. Man-Machine Systems R&D 
8. Selection/Classification 
9. Applied Systems Development 
10. Career Structures 
11. Personnel Retention 
12. Attitude/Motivation 
13. Minority Discrimination Measurement and Reduction 
14. Organizational Effectiveness 
15. Distribution/Assignment 


16. R&D Resource Management 
17. Recruitment/Procurement 
18. Personnel Performance 


B. PROCEDURE TO QUANTIFY THE DATA 


These responses were processed using the Ford procedure 
as adapted for use on the IBM 360/67 computer at the Naval 
Postgraduate School... This procedure permits each judge to 
rate only those alternatives which he feels competent to 
rank, to set his own ranking scheme, and to make use of ties 
for alternatives deemed equal. The procedure then takes 
these qualitative ordinal rankings by all the judges within 
that panel and assigns a quantitative weight to each item in 
the set being ranked. 

It does this by forming a win-loss matrix A = (ais), 
where aig represents the number of times program element i 


was preferred to program element j. If any of the program 





elements were judged universally high-preferred to all other 


program elements, or universally low--all other program 
elements being preferred to it, by any panel that program 
element would be removed from the matrix and so labeled. A 
weight, w 


ay L@ them determined ateratively for each of the 


program elements remaining in the matrix using the equation 


nn Jn 
Wi + Ws 


where n is the Auer of iteration. The weights, wy and 
we, on the first iteration are the win percentage for program 
element i with respect to program element j for all j, all 
Sionents in the matrix. The iterations stop when an assigned 
convergence/stabilization criteria for all the weights are 
met or when a specified number of iterations have been 
completed; in this analysis the convergence criteria was a 
change of less than 0.005 and the limit on the number of 
iterations was 50. The set of all these weights is the 
maximum liklihood of obtaining the original win-loss matrix 
(less any universal highs or lows). Any individual weight is 
then the probability or “odds", wy , that program 

wyt W5 


element i is preferred to program element j. 





C. DESIGN OF THE MODEL 


The inputs to any particular personnel research project 
are professional man years, professional costs, special costs 
and contract costs; budget allocation was used as a proxy 
variable for the summation of these costs. Since these costs 
are not hard (such as weights, size, number, power, et 
cetera) and the decisions on which personnel research projects 
to fund and at what level are judgmental, a parametric model 
.was deemed most appropriate in an attempt to analyze the 
opinions expressed with respect to the Navy Personnel Research 
Program in support of GOR 43. 

The two criteria, military significance and investment 
risk, of a program element which were ranked by the judges 
were designated independent variables. The weight assigned 
by the Ford procedure to a criterion of a program element was 
used as an observation for that variable by a panel for that 
program element; this resulted in 18 obervations per panel 
per variable. Under the assumption that the overall value of 
a program element was some function of these variables, 
Multiple regression could be used to test the fit of data 
to various functions and thereby determine which function 
would produce a consistent model with which to compare data 
for each panel of judges. The 1973 fiscal year budget allo- 
cation for each program element was selected as the dependent 
variable for these regressions. The rationale for this was 


the amount of dollars allocated to a program element was a 


10 





measure of its assumed worth, value or utility to the 


decision-makers; as such, it seemed reasonable that the 
amount of money allocated to a program element (C) would also 
be a function of its military significance (M) and its 
investment risk (I): C = £(M,I). 

Unlike many multiple regression models, this model was 
not to be used to estimate parameters, the coefficients, nor 
to predict the dependent variable, budget allocation. This 
model was developed to compare the data produced by the 
rankings of the nine panels and thereby indicate the amount 
of agreement/disagreement among the panels and to indicate 
the contributors to this dLliterence. 

An indirect application of the model was used for compar- 
ison of the individual program elements. This was necessi- 
tated because there is only one observation for the dependent 
variable, C, the amount of budget allocation to a program 
element. 

The measure of effectiveness selected for the analysis of 
these regressions was the coefficient of determination, R2, 
this cociilicient 1s destined to be the ratio of the sum-or 
squared deviations about the mean due to the regression to 
the total sum of squared deviations about the mean, 


bixt xb 


Rv = 7. , where b is a vector of parameter 
YY ° 


coefficients, X is a matrix of observations of the independent 
variables and Y is a vector of observations of the dependent 


variables. Since “the coefficient of multiple determination 


LL 





may be considered as showing the improvement of the closeness 


of the regression plane to the actual points relative to the 
fit of the plane going through the mean"4, the difference a 
this coefficient for the regressions with data from the nine 
individual panels was used as the relative measure of 
difference between the panels. 

If the distribution of the dependent variable, budget 
allocation of a seoguatt element, for the two given independent 
variables is from a multivariate normal distribution or if 
this assumption is imposed, more powerful measures of effec- 
tiveness are available and stronger statistical tests and 
conclusions made. The normal assumptions were not used in 
this analysis because (1) the assumptions seemed to be too 
restrictive since the analysis was comparative eng and 
(2) with the data base limited to observations from one year, 


the assumptions could not be readily justified. 


4 Yamane, Taro, Statistics, An Introductory Analysis, 
Harper-Row, 1964, p. 652. 
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Il 


A. THE DATA QUANTIFIED 


Of the 263 questionnaires mailed, 


I. RESULTS 





198 responses, 74.5 


percent, were returned; Table III presents the responses per 


TABLE IIT 


Survey Response 


panel. 

Panel Questionnaires 
100 | 18 
200 30 
300 18 
400 30 
500 32 
600 30 
700 75 
800 9 
900. _22 

TOTAL : 263 


16 
16 
18 
28 
28 
28 
44 
4 
_16 


oy 


Responses 


Percent 


76.2 





74.5 


The weights assigned by the Ford procedure for military 


Significance and investment risk are presented in Tables 


Iv and V respectively. Table VI presents the program 


elements, the rank of their allocated budgets, the mean weight 


for military significance and its rank, and the mean weight 


for investment risk and it 


s rank. 


13 


VT 


Program 


Element 


WOoOwAINUO BWNY EF 


Panel 


100 


0.34227 
Le2ilas 
0.4134 
0.4705 
L«0103 
0.9275 
Detar 
0.3319 
0.1456 
0.2044 
Lo 3.L3 
0.2444 
0.3190 
0.2091 
0.2926 
0.1586 
I.90G5 
0.4339 


oooocnooqcneo°o°oco 
. 8 © © e 8 © © © @¢ @ 


TABLE IV 


WEIGHTS ASSIGNED MILITARY SIGNIFICANCE 


Panel 


200 


° 
Ay 
CO 
Ww 
(oe) 


Panel 


300 


~4342 
~7113 


Panel 
400 


0.2977 
2.7825 
6.3121 
0.2022 
0.2957 
0.2943 
0.2705 
0.9514 
0.2648 
0.2360 
0.9201 
0.5931 
0.2362 
0.3798 
0.3216 
0.3846 
0.8820 
0.7894 


Panel 


500 


0.2448 
0.7789 
Q+301l2 
0.3962 
0.8706 
0.5927 
O.59 94 
0.5166 
0.4215 
0.3483 
1.0004 
0.3601 
0.1370 
0.5379 
0.2832 
0.3489 
0.5079 
0.9786 


Panel 


600 


Osl728 
3.0687 
0.3510 
0.3060 
0.4554 
0.4526 
O),.2998 
0.7471 
0.3642 
0.2641 
0.6154 
0.2420 
Oe lal 
04.3336 
Opeaa1 
0.2602 
0.5625 
LedQ39 


Panel 


700 


Vesl7s 
Owed oh? 


oO 
Ny 
Ol bk 
U1 Ww 
N Nh 


-6076 
- 3570 
23 L236 


uo 
NO 
\Oo 
oO 


-2780 
-6285 


~2499 
-2174 
~4942 
-4278 
«2961 


0.7824 


ooooococorcc0c coc 0 
[o>) 
nN 
Oo 
Ay 


m1 
~~ 
co) 
\o 


Panel 
800 


0.2099 
Lee L 


oO 
e 
Ny 
RM 
~I~ 
CO OV 


oe bes 
- 4628 


= 
(oe) 
NO 
Ne} 


eee 
-2376 
-5566 
-2404 
«9036 
-2156 
-9856 
3437 
22203 
-5914 
-4483 





Panel 


900 


0.3514 
1.6246 
0.2469 
0.0552 
0.2545 
0.2955 
0.4940 
0.4548 
0.4876 
0.4982 
0.7781 
0.7141 
0.4730 
0.5600 
0.5355 
0.1935 
0.5419 
1.0378 


ST 


Program 


Element 


WwOaANNDUBWN EE 


Panel 


100 


0.2355 
lerJ30 
0.9565 
O.6705 
0.6967 
Le VL0Z 
0.2189 
0.6456 
1.1416 
0.4240 
0.2891 
0.1587 
0.1276 
0.0722 
0.6910 
On 2832 
0.4412 
0, 1673 


0 
0 
0 
0 
0 
0 
0 
0 
1 
0 


oqoooooo0°co 


TABLE V 


WEIGHTS ASSIGNED INVESTMENT RISK 


Panel 


200 


» 0910 
«9780 


.6985_ 


-5622 
16736 
-6461 
earo0 
a5 796 
~6861 
-4654 
-4320 
- 1428 
Lay 3 
-3328 
-6014 
-2869 
. 7800 
23541 


Panel 


300 


0.2247 
0.919% 
1.1836 
0.6969 
0.7160 
0.3806 
043819 
0.6180 
1eS7L9 
0.8144 
OG. 2972 
Q.1372 
0.1156 
0.2163 
G6 105 
0.1347 
Os.2001 
0.3629 


Panel 
400 


0.2049 
1.8326 
0.4809 
«GoL/ 
-6524 
e2975 
~4391 
25335 
0996 
-4701 
- 3099 
~2431 
«2016 
-2106 
~8512 
~3743 
-6297 
0.5013 


oooaoooccocooo°oo°o 


Panel 


500 


0.2451 
0.6306 
0.6380 
0.8054 
0.8601 
0.4955 
0.7491 
0.25525 
iQ LD 
0.2932 
0.4549 
0.22541 
0.1448 
0.4267 
0.3301 
O<2519 
0.4938 
0.6985 


Panel 


600 


GaL33 1 
2.2000 
0.4633 
G.7511 
0.4952 
0.4755 
0.4938 
0.9254 
0.7562 


° 0.4482 


0.2130 
Os llis 
042136 
G.1662 
1.0304 
042967 
O23 Lee 
0.6207 


Mootle hohe ke hele le tee e eee eee 


Panel 


700 


-4856 
«e822 
-5270 
-5469 
-5090 
2631 
-4220 
-6942 
-4386 
-4400 
25475 
.4403 
- 3436 
3148 
.7057 
s2908 
~4229 
af oo 


Panel 


800 


0.1924 
0.4608 
0.4452 
0.7789 
0.9703 
0.3196 
1.5606 
0.4791 
0.8807 
0.4085 
0.3782 
0.1422 
0.0829 
0.2768 
0.3606 
0.3414 
O2.8563 
L.3082 





Panel 


Ne) 
oO 
oO 


0.1983 


HD ov WO 
mW WO 
aOanoW 
CO ww 


> 
Oo 
fee) 
(on) 


«2790 
.9416 


3604 
62393 


ies) 
ioe) 
foe) 
oO 


~1814 
sotge 
-2965 
goo? 
0.6228 


coocooocodoaocccocoo°c°oo 
em ~ 
\O Fe 
of 
NM 


oT 


PROGRAM 
ELEMENT 
1. Drug Abuse 
2. Educ. & Training 
3. Occup. R & D 
4, Man. Cost Sys. 
5. Man. Rgmts. 
6. Pers. Mgmt. Sys. 
7. Man-Mach. Sys. 
8. Sel./Class 
9. Applied Sys. 
10. Career 
ll. Retention 
12. Attitude 
13 Minority 
14. Organization 
15. Distribution 
16. R & D Resources 
17. Recruitment 
18. Performance 


PROGRAM ELEMENTS, BUDGET RANK, ME 


BUDGET 


RANK 


TABLE VI 


MEAN MIL. 


SIG. WEIGHT RANK 


0.2753 
1.3867 
0.4527 
0.3115 
0.7115 
0.4828 
0.3835 
0.62390 
0.3745 
0.3927 
0.9948 
0.4628 
0.2303 
0.4733 
0.4031 
0.2583 
0.8139 
0.7405 


AN MILITARY SIGNIFICANCE 
AND RANK, MEAN INVESTMENT RISK AND RANK. 
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1 
10 
15 


MEAN INV. 





RISK WEIGHT RANK 


0.2234 
1.1393 
0.6359 
0.6845 
0.6905 
0.4896 
0.6244 
0.6190 
1.0223 
0.4727 
0.3536 
0.2083 
0.2028 
0.2442 
0.6178 
0.3116 
0.5437 
0.5466 


kK 


POI ARPoO BOE HN 





Table VII presents a summary of the criterion ranked 


higher. A majority of the nine panels of experts ranked the 
investment risk of nine of the program elements higher than 
the Nica Gary Significance, one program element Applied 
Systems Development, was ranked highest by all groups. Eight 
program elements were ranked higher in military significance; 
three were so ranked without any dissent: Personnel 
Retention, Attitude/Motivation, Organizational Effectiveness. 
The nine panels were split in their ranking of one program 
element, R&D Resource Management, four panels ranked it 
higher in military significance, four ranked it higher in 


investment and one ranked it about equal in each. 


17 





TABLE VII 


CRITERION RANKED HIGHER 


Program Elements Ranked Higher in Investment Risk 


Occupational R & D 
Manpower/Personnel Cost Systems 
Personnel Management 
Man-Machine Systems R & D 
Selection/Classification 
Applied Systems Development 
Career Structures 

Minority 
Distribution/Assignment 


Program Elements Ranked Higher in Military Significance 


Drug Abuse 

Education and Training 
Personnel Retention 

Attitude Motivation 
Organizational Effectiveness 
Recruitment/Procuroement 
Personnel Performance 
Manpower Requirements 


Program Elements Ranked Equal in the Two Criteria 


R & D Resource Alanagement 


18 





These results seemed to indicate a certain degree of 


agreement among the panels about the relative position of a 
program element or the two criteria, in that either investment 
risk or military significance of all but one element was 
assigned a higher weight by a majority of the panels of 
experts. Additionally, the ranks of these weights for the 

18 program elements by the nine panels for each of the two 
criteria indicated some degrees of agreement/disagreement 
among the: panels; Tables VIII and IX present the rank of the 
assigned weight of a program element by a panel for each of 


the two criteria. 
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- PITLE 
1. Drug Abuse 
2. Educ. & Trng. 
3. Occup. R & D 
4, Man. Cost Sys. 
5. Man. Rgmts. 
6. Pers. Mgmt. Sys. 
7. Man-Mach Sys. 
8. Sel./Class 
9. Applied Sys. 
10 Career 
11. Retention 
12 Attitude 
13. Minority 
14. Organization 
15 Distribution 
16 R & D Resources 
17. Recruitment 
18. Performance 
* 
element. 


TABLE VIII 


RANK OF PROGRAM ELEMENTS BY THE 


WEIGHT ASSIGNED MILITARY SIGNIFICANCE 


ee Cee eee 


10 


18 


17 


11 


10 


16 


12 


11 


20 


13 


15 


17 


Rank of the average (mean) weight 


17 


16 


13 


18 


13 


17 


10 


13 


12 


10 





16 


1 


10 


15 


assigned to a program 


TABLE IX 


RANK OF THE ELEMENTS BY THE 
WEIGHT ASSIGNED INVESTMENT RISK 


‘h 


TITLE 100 200 300 400 500 600 700 800 900 AVG 
1. Drug Abuse 13 18 24 18 16 17 9 16 17 «16 
2. Educ. & Trng. 1 8 3 1 5 1 1 8 1 1 
3. Occup. R & D 4 3 2 9 7 10 7 9 #15 5 
4, Man, Cost Sys. 7 10 6 4 3 5 6 6 4 4 
5. Man. Rgmts. 5 4 5 3 2 7 8 3 7 3 


6. Pers. Mgmt. Sys. 3 5 120 dz Ae 6 15 d4 it dt 


7. Man-Mach. Sys. 14 9 9 11 4 8 14 a 2 6 

: 8. Sel./Class 8 7 7 7 8 3 4 7 9 vi 
: 9. Applied Sys. > £° 4 © 2 AD 4 3 2 
10. Career 100611 2 Vo (2 Ty. dl 20 10 22 

11. Retention tis 22 Gas a 20 ds 5 1h 2a: 13 

12. Attitude 1¢ Ty Ie 26 15° 18 20 7 126.1? 

13. Minority i‘? 26 2 GS We 14 26 28, 22 12 

14. Organization if a4 5 17% 21. 26 27 25 de 

15, Distribution. 6 6 8 2 13 2 2 22 13 8 


16. R&D Resources 12 15 17 13 #17 #13 #18 #13 8 14 
17. Recruitment 9 2 12 5 9 12 13 5 6 10 


18. Performance 15 13 #141 8 6 6 2 2 5 9 


* Rank of the average (mean) weight assigned to a program 


element. 
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’ B. MODEL SELECTION 


The weights assigned to military significance, M, and 
investment risk, I, by each of the panels were ordered by 
their rank and the weights plotted, as were the budget 
allocations for program elements, C. Since program element 
numbers 1-18 were randomly assigned initially the order of 
the data had no effect; plotting the data suggested various 
functions to test for their fit. Several linear and log- 
linear functions were investigated with the data in a search 
for the most consistent model for C = £(M,I). These are 
listed in Table X, each was tested with and without the 


constant, bo: 


TABLE X 
Parametric Models Tested 
C= Bo + bjIM + Bol +e 
C= be + byM + bgt + bgMal +e 
C = bo + bIM + boI + b3M/I +e 
C} be + bYM + bol + b3I/M +e 
C= by + MblTo2 4 © 
C = boMP11>2m*rb3 + © 


C = bomP1rP2m/1P3 + 


C = bomPirb2r/MP3 + e 
* Note: In these models, C = (Cy), where C; is the budget 
allocation for program element i; M = (Mj), where M is the 


weight assigned the military significance of program element 
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i, I = (Iz), where I is the weight assigned the investment 


risk of program element i; i =1, 2, ..., 17, 18. 

The investigation indicated that the simplest two- 
variable model, C = bo + b1]M + b2I, gave the most consistent 
results and the best fit for the data. This model was 
developed assuming that the partial derivatives of the function 
with respect to each of the variables M and I to be positive 
and to reflect that (a) the greater the military significance 
the more money will be allocated to that program element and 
(b) the better the investment Hie the more likely money 
will be allocated to that program element. Using linear 


equations to express this mathematically, the differential 


equations: 
lef =b , b o 
= 1 i. 
2)£ =b» bes 
2 = 


yield the above equation as the solution to C = £(M,I). 


C. COMPARISONS OF PANELS 


This model was then used with the FY '73 allocation and 
the weights assigned to M and I by the Ford procedure for 
each of the nine panels. The step-wise regressions were 
computed using BIMEDO2R as programmed at NPS; Table XI 
presents the final results for each of the nine panels. 
There were no indications of multicollinearity or hetereo- 
scedasticity in any of the regressions. The difference in 


the coefficients of determination, R?, is a relative 
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measure of the difference between the panels over the budget 


allocated to Navy Personnel Research programs in support of 


GOR 43. 
TABLE XI 
Regression Results for All Program Elements 
Panel R? E 
100 - 6206 126267 
200 -0403 0.615 
300 ~1210 1.032 
400 -8896 60.422 
500 .1065 0.894 
600 -8369 38.486 
700 ° .4094 11.089 
800 ~1774 Leh? 
900 26591 14.499 


These results seemed to indicate a reasonably good fit 
for five of the panels (100, 400, 600, 700, 900)°, but not 
for the remaining four. Under the assumption that this 
represented a possible demarcation line of agreement and 
disagreement within the community, the equality® of all 
possible pairs of regressions were tested to determine if the 
two regressions came from the same population, under the null 


2 If the normality assumptions are imposed, these five 
F-statistics are significant for the null hypothesis 
Ho: 8 = 0.0, with q = 0.10. 


6 
p. 147. 


Thiel,. Henri, Principles of Econometrics, Wiley, 1971, 
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hypotheses Bis Bo: versus the alternative hypothesis, 
B 1 # B 9. Ata = 0.05, none of the Held WeSenesed could 
be rejected; thus, it was concluded that all of the panels 
came from the same population. 

' An aggregated eberetotent of determination was derived 
by regressing 162 observations (9 panels x 18 observations 
per panel) for a consolidated panel, all in the model. This 
coefficient can be compared to that for an individual panel. 
Additionally, nine regressions with data from eight of the 
panels were computed. This changed the aggregated coefficient 
of determination, reflecting the contribution of the excluded 
panel's observations, thus providing another means of com- 
parison.’ These results are presented in Table XII. As 
expected, excluding a panel which had a reasonably good fit 
decreased R2, whereas excluding a panel which did not fit 
the model will increase this coefficient. 

TABLE XIT 


Regression Results Excluding One Panel 





‘Panel RZ PF 
all 3362 40.235 
100% 63237 33.747 
200* 3796 43.128 
300* 3729 49.279 
400* .2719 26.6322 
500* .3657 40.643 
600% 27a? 26.696 

700% .3575 39.222 
800* .3711 41.596 
900% .3194 33.083 


* panel excluded 





7 The rationale for this will be discussed in Section 
Iv. 
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Since there was a certain relationship between some of 


the panels, such as between the Managers and Professionals at 
NAVPERSLAB, SDGO, and possible relationships among stners, 
data for these pairs of panels were regressed in the model. 
These results are presented in Table XIII; Table XIV 
presents the results of regressions of the data from the other 
seven panels. 

TABLE XIII 


Regression Results for Panel Pairs 








Panels R F 
All . 3362 405235 
100 & 200 .2792 6.392 
300 § 500 + .1032 1.899 
400 & 600 .8460 90.633 
300 & 600 ~4505 13.527 
400 & 500 .4841 15.483 

TABLE XIV 


Regression Results Excluding Panel Pairs 








Panels R F 
All e302 40.235 
100 & 200 «9708 364302 
300 & 500 -4149 43.604 
400 & 600 1522 14.642 
300 & 600 xol23 27. 93oL 
400 & 500 «2963 255890 
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D. COMPARISONS OF NAVPERSLABS 


Since the majority of this research is done in the two 
Naval Personnel Research and Development Laboratories, one in 
San Diego and the other in Washington, D.C., the program 
elements conducted primarily under their auspices® were 
regressed in the model. Table XV presents these results for 
the consolidated panel and each of the individual panels for 
the Washington laboratory program elements; Table XVI, the 
results for the San Diego laboratory. 

TABLE XV 


Regression Results from Program Elements 
Under Cognizance of NAVPERSLAB, WASH, D.C. 


Panel R? F 
All - 1308 7.224 
100 - 3030 1.739 
200 - 1840 0.904 
300 »3238 1.915 
400, ~2209 tid od 
500 . 20361 0.150 
600 ~2041 1.026 
700 ~5154 4.254 
800 ~0505 0.021 
900 2617 1.418 

8 


Reference 1. Program elements 1, 3, 4, 51 7, 9, 10, 
12, 14, 16 and 17 are to NAVPERSLAB, WASH, D.C.; the other 
to NAVPERSLAB, SDGO. 
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TABLE XVI 





- Panel Ro 

All 4827 

100 7864 

200 0371 

300 6626 

400 9614 

500 4239 

600 9348 

: 700 4250 
: 800 2064 
900 6578 
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Regression Results from Program 
Under Cognizance of NAVPERSLAB, SDGO. 








E. COMPARISONS OF FUNCTIONAL AREAS 


The program elements in the Naval R&D Program are 
separated into the six functional areas” listed below. The 
numbers in parentheses coincide with the numbers assigned 
program elements in that area. 

Manpower Resources Management (3, 4, 5, 10) 
Personnel Operations R&D (6, 8, 11, 13, 15, 17, 18) 
Organizational Management (1, 12, 14) 

Human Factor Engineering (7, 9) 

Education and Training (2) 

R&D Resource Management (16) 

Regression results from the first two areas are presented 
in Tables XVII and XVIII respectively. Results for the 
other areas are not directly available from this model 
Heese Caution must be used when referring to these 


tables since the number of observations are small. 


9 Bureau of Naval Personnel R&D Program FY '73. 


10 his will be discussed in Section IV. 


As 


' Panel 


All 
100 
200 
300 
400 
500 
600 
700 
800 


900 


Panel 


All 
100 
*200 
300 
400 
500 
600 
700 
800 


900 


Regression Results from Program Elements 
jn the Manpower Resources Management Functional Area 


Regression Results from Program Elements 
in the Personnel Operations R&D Functional Area 











TABLE XVII 


SS Spe 


—_. 





5.054 
0,132 
1.233 
0.070 
0.462 
4.594 
e091, 
6.112 
380.440 


50.892 


TABLE XVIII 


F 


12.640 
2.488 
5.193 
4.180 
1.864 
4.889 
0.533 
0.180 
tei 
0.192 
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F. COMPARISONS OF PROGRAM ELEMENTS 


The model selected did not provide direct comparisons for 
the individual program elements since the data consists of 
FY '73 observations only.11 an indirect method utilizing 
the model was attempted; this approach is the same as that 
in Section III E when one panel's observations of all 18 
program elements were excluded from the composite panel of 
all nine panels. This method was suggested by the Jackknife 
procedure for bias reduction [3] and its resultant pseudo- 
values. Each panel was regressed using 17 program elements, 
excluding a different element each time. A pseudo-coeffi- 
cient of determination, P, for the regression without one of 
the program elements was thus provided. 

The difference between the coefficient of determination 
and the pseudo-coefficient indicated whether the excluded 
element had a positive or negative contribution to the 
overall regression. A positive contribution occurred when 
an excluded element produced a lower pseudo-coefficient of 
determination, and a negative contribution occurred when 
the pseudo-coefficient was higher. Mathematically the 
positive contribution occurs because the excluded element 
produces a greater contribution to the computed sum of 
squares than it does to the actual sum of squares; whereas 


a negative contribution produces a lesser contribution to 


di This will be discussed in Section IV. 
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the computed sum of squares than it does to the actual sum 


of squares. 

The percentage amount of this contribution, (P - R2) /R2, 
indicated the difference between the panels for the program 
element excluded. R2, P and this percentage are presented in 


Table XIX for the 18 program elements. 
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TABLE XIX 


R2, P, 3R 

490 5900 
3396 .1065 
8891 .0920 
= 0.1 a1 6 
2056 0871 
~76.9 =i, 2 
3928 .1065 
0.4 0.0 
8935 1122 
0.4 54 
8897 ora 
0.0 19.6 
po75- .2067 
0.9 0.2 
8912. £1314 
0.2 a3 4 
9044 1073 
0.5 0.8 
8966 1649 
0.8 Se Pe 
9143 .1204 
2.7 13.4 
.8918 aoe 
0.2 34.7 
9067 isi 
0.8 8.3 
.8865 .O811 
= 043 -24.5 
.8877 1077 
a 12 
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TABLE XIX (Cont'd) 
400 500 

9023 LO6L 

V3 - 0.4 
8940 1918 
- 0.5 -S2.4 
8964 - 1063 

0.8 m2 
8941 -1392 

0.5 31.6 


Number of Program Element Excluded 

















IV. DISCUSSION 


The overall response, 74.5 percent, was considered 
sufficient to svopade an adequate data base for the analysis. 
The response per panel, as shown previously in Table III, 
seemed ample as to be representative, with the possible 
exception of Panels 200, 700 and 800. The sensitivity of the 
Ford procedure to an individual judge is the ratio of the 
number of pairwise comparisons he ewe to the total number 
of comparisons made by his panel, thus, the more program 
elements ranked, the more ranks used and the fewer ties 
allowed, the more the influence of that judge. 

In all the ee except 200, 700 and 800, the number 
of total comparisons was such that the most influential judge 
contributed about eight percent as compared to an expected 
mean contribution of 6.25 percent; whereas the least influ- 
ential contributed four percent. The representability of 
the Ford results were further verified when, prior to the 
final computations, the sizes of three of the panels were 
increased and the resulting change in the assigned weights 
were small. | 

The weights assigned by panels 200, 700 and 800 however, 
are subject to significant change with an inclusion of 
additional responses. Accordingly, these panels were 
included in the analysis for completeness, but were not 


accepted as representative. 
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Three criteria were used in the evaluation of the 


en cl geet of each of the multiple regression models 

tested. These were whether (1) the signs of the parameter 
coefficients, b, and bg were similar for each set of data, 
(2) these coefficients were intuitively reasonable with their 
mathematical derivation, and (3) each step in the regression 
produced only modest and logical changes. 

The model thus selected is comparative only; it does not 
mean that this function completely describes, nor best 
fits the determination of the Navy Personnel Research 
Program in support of GOR 43, and this was not its purpose. 
Additionally, it does not mean that the panels whose data 
fit the model "best" are those whose judgments are "best" 
or correct. 

Tables XI and XII show that indeed the model did provide 
results which indicated a substantial difference between 
panels of the personnel research community in their rankings 
of program elements. This was reftented primarily by the 
difference in the cont iited outs of determination but also by 
the difference in the parameter coefficients as well as in 
the oe and standard deviations of weights assigned 
military significance and investment risk. There was a 
distinct split of the community into two groups; those whose 
data fit the model reasonably well and those whose did not 
fit the model well. The results also reflected a difference 
of the two panels of the Department of the Navy located in 
Washington, D.C., a difference in the two NAVPERSLABS-- 


both from the managers' rankings and the professionals' 
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rankings, and yet a similarity in the rankings of the profes- 


sionals and managers for the same NAVPERSLAB. The results 
in Tables XII and XIV substantiated this finding. 

The results from the model when it was applied to work 
at each of the two NAVPERSLABs, Table XV and XVI, indicated 
significant differences between the work at the laboratories, 
and relatively more agreement between panels. In general, 
the model results indicated a reasonable fit for the program 
elements at San Diego, but not at Washington, D.C. 

It seems noteworthy to mention that data from the two 
panels at NAVPERSLAB, WASHDC provided distinctly different 
results for work at their laboratory, and that both their 
rankings of program elements at NAVPERSLAB, SDGO provided a 
substantially better fit to the model. The data from both 
panels at the San Diego laboratory provided results con- 
Sistent with the rest of the community for program elements 
at Washington; whereas the results for their own work were 
higher than the rest of the community. 

The results from the model when it was used with data 
from Sie functional areas distinguished differences between 
the panels. These indicated areas/program elements from 
whence these differences stemmed. 

In the Manpower Management area, all the program elements 
are work from NAVPERSLAB, WASHDC, as are the program 
elements in the area of Organizational Management. 

Comparing the results for each panel from Table XVII 
indicate a generally better fit in the Manpower area; the 


exceptions are panels 100 and 300. Furthermore, the results 
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for the two panels from the Washington laboratory are signif- 


icantly different in this area. This assists in the 
explanation for their overall difference previously mentioned. 

The program elements in the Personnel Operations area are 
assigned to NAVPERSLAB, SDGO; their only other element is 
number two, Education and Training. Comparing the results in 
Tables XII and XIV, there is a better fit in the Personnel 
Operations area for panels 200 and 800, and a slightly 
better fit for panels 300 and 500. There is a significant 
difference in the fit for the two San Diego panels within 
this area, as well as- between the two areas. This indicated 
that Program Element Two contributed Substantially to the 
fit. Bor al. program elements at their laboratory, as well as 
that there is some disagreement between the two panels in 
the Personnel Operations area. 

The results from the model for comparison of the individual 
program elements were not strong since the data base consisted 
of observations for only one year. Previous years budget 
data were mane but judgments from earlier years for 
the independent variables, M and I, were not. fThus, although 
there are — observations, one for each panel, for M and I, 
there is only one observation for C; hence C, the amount of 
budget allocation to a program elements, does not vary 
linearly with changes in military significance and investment 
risk as hypothesized by the model. Accordingly, application 
of the model with this data results in a regression which 


states that C equals the constant, b,, and that by and b9 are 
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zero. Future repetition of the survey would provide the 
saad tional data necessary for application of the model 
directly with individual program elements. 

The indirect application of the model provided a method 
which allowed comparisons of an individual program element 
between the panels and between program elements within a 
panel. These results coincided with results from direct 
application of the model. For example, as previously 
“mentioned, the two NAVPERSLAB, WASHDC panels differed 
distinctly over work on program elements at their laboratory, 
and Table XIX not only supported this difference but 
indicated the program elements where the difference occurred; 
similarly for their comparisons in the functional areas. 
Comparisons of the two San Diego panels substantiated 
the earlier claim that Program Element Two, Education and 
Training, contributed significantly to their overall fit 
with the model. 

Using the percentage amount of contribution per program 
element as a measure, the results from all nine panels were 
unanimous for three elements; Drug Abuse (1) provided a 
papiittaars contribution, Manpower/Personnel Cost Systems (4) 
and Personnel Management Systems (6) provided negative 
contributions. The results for three other elements 
indicated they provided negative contributions: Occupational 
R&D (3), Manpower Requirements (5) and Attitude/Motivation 
(12) The remaining 12 elements provided positive contributions 


to some panel's regressions and negative to others. 


a9 


V. CONCLUSION 


The Ford procedure provided an acceptable, economic and 
efficient method of quantifying the survey responses. One 
hundred percent response to the survey would be necessary 
to ensure the weights assigned by a panel completely reflected 
that panel's rankings. However, since the sensitivity to an 
individual response decreases as the proportion increases, 
six of the nine panels' responses were deemed representative 
for this analysis. 

The selected model emogiedd results sufficient to compare 
the difference between the data from the panels as pertains 
to the overall Navy Personnel Research Program in support of 
GOR 43, the program elements at the two laboratories, and 
the functional areas encompassing more than two program 
elements. tthe results from application of the model to these 
separate tasks substantiated and supplemented the individual 
results from a single task. The consolidated results manifest 
these differences. Extension of the model for parameter 
estimation, prediction or evaluation of single program 
elements must remain tentative until a more extensive data 
base can be collected. 

This model and analysis treated recommended changes to 
the Navy Personnel Research Program exogenously. To do 
Otherwise would have required value judgments as to which 


panels' rankings were better and by how much, as well as 
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validation of the selected model as deterministic. These 


were both beyond the scope of the present work, but the 
results herein should be of assistance in recommending 


changes. 
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APPENDIX A 


SAMPLE QUESTIONNAIRE 


‘MILITARY SIGNIFICANCE AND INVESTMENT RISK OF 
' PERSONNEL AND TRAINING RESEARCH PROGRAMS 


1. GENERAL 


Here are 18 descriptions of program elements in the Navy's 
Personnel Research Program. Please rank them twice according 
to two different criteria. The first time, rank them 
according to their military significance. The second time, 
rank them according to the investment risk in committing Navy 
resources to them. : 


a Criterion 1 


Military significance refers to the needs of the 
Navy. Rank from most to least the importance of problems that 
lie in the area of each program element. Indicate where the 
Navy would obtain the greatest payoff given that the research 
is successful. 


b. Criterion 2 


Investment risk refers to the probability of a 
successful research effort independent of the value of the 
payoff (Criterion 1). Rank these elements from good to bad 
for the probability that the research will be successful. A 
research effort is successful if solutions to problems--or 
significant improvements--can be developed within a period 
of approximately ten years. 





ay RULES 

a. You need not rank all elements for each criterion-- 
only those you have confidence in ranking. You need not rank 
the same number of elements for each criterion. 


b. Use as few (3) or as many (18) ranks as you desire. 


c. Ties (up to six) are permitted within any rank you 
set up. 


ad. A rank of one is the greatest in military 
significance and the best in investment risk. 
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3. SUGGESTED APPROACH 


Do the rankings separately for each criterion. The 
following appraoch is suggested: 


a. Separate the element descriptions into those you will 
and will not rank. 


b. Separate those you rank into broad hierarchical 
categories for a start. 


c. Try to break down the categories into finer 
discriminations; continue until you cannot break down any of 
the categories further. 


d. Consecutively assign numbers to the categories from 
one to as many categories (ranks) as you have. 
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01 DRUG ABUSE 


This area consists of research and development for 
(a) determining causes, attitudes toward, and extent of drug 
abuse in the Navy; (b) exploration of means to reduce, prevent, 
and control drug abuse; and (c) means to provide effective 
education, counseling, and rehabilitation programs for Naval 
personnel and dependents. 


Typical Projects 


1) Early Identification of Drug Abusers and Potential 
Drug Abusers 


2) Methodology for Reducing Drug Abuse 


02 EDUCATION AND TRAINING 


The two mutually supporting purposes of Education and 
Training research and development are (1) improving Navy 
training and (2) development of principles and technology of 
training. Navy requirements call for more accurate definition 
of training methods directed toward meeting future job 
requirements. Research and development in this area involves 
work on the broad spectrum of problems related to deriving 
and specifying appropriate training content and developing 
and evaluating ways to improve acquisition of training 
objectives. Research in this area also includes specific 
problems in learning, perception, job aids, programming, 
displays, simulation and management as they relate to 
improvement of training. 


Typical Projects 
1). Computer Administered Instruction 


2) Computer Managed Instruction 
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03 OCCUPATIONAL RESEARCH AND DEVELOPMENT 


Development, test, and evaluation of new and improved 
techniques for describing, evaluating, structuring, and 
re-engineering Navy occupations for optimal matching of 
available manpower characteristics to changing job require- 
ments. Involves new methods for determining skills required 
for effective performance of occupations and the storage 
and retrieval of occupational data pertaining to work 
requirements of Navy Personnel. 


Typical Projects 


1) Design of Navy Occupational Analysis System and 
Occupational Data Bank for Officer and Enlisted 


2) Development of Optimum Enlisted Classification 
Structure 


04 MANPOWER/PERSONNEL COST SYSTEMS 


Development, test, and evaluation of new and improved 
(a) techniques for the acquisition, computation, storage, 
and retrieval of manpower/personnel cost data; (b) techniques 
for determining the costs of various mixes of men and equip- 
ment, personnel management policies, and personnel utilization 
factors; and (c) measures which permit manpower requirements 
planners to trade off cost and effectiveness in order to 
evaluate available options. The principal objective is to 
furnish total government costs from procurement through 
post-separation which can be used to formulate cost-related 
decisions concerning the total Navy manpower personnel 
system. 


Typical Projects 


1) ‘Development of Cost Techniques for Use in Determining 
Manning Levels 


2) Budget Cost Management for Personnel Planning 
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05 MANPOWER REQUIREMENTS 


Development, test, and evaluation of techniques for 
defining and displaying quantitative and qualitative manpower 
requirements for all functions performed ashore and afloat 
including upward aggregation of requirements to the total 
naval establishment. The techniques must be responsive to 
changing operational requirements, system design innovations, 
and available human resources. Includes the development of 
manpower/productivity models and the use of simulation 
techniques to test alternative manning and manpower management 
concepts in terms of cost and total system effectiveness. 

The principal objective is to develop methods for determining 
Manpower requirements which are capable of responding quickly 
to changes in resources and military objectives and 
requirements. 


‘Typical Projects 
1) Manpower Planning and Allocation Techniques (Ashore) 


2) Reduction of Manpower Requirements Afloat 


06 PERSONNEL MANAGEMENT SYSTEMS 


Development, test, and evaluation of tools to improve the 
quality of personnel management by increasing the accuracy of 
force projections, by responding more rapidly to changing 
policies and operational conditions, and by projecting the 
effects of current and proposed personnel policies to deter- 
mine the most effective personnel management strategies. 
These tools are usually in the form of computerized models of 
the personnel system which can be used to reflect the 
relationship between such variables as recruit input, advance- 
ment, and attrition and to project the effects of these 
variables upon the size and skill composition of the 
personnel force. 


Typical Projects 
1) Development of Advanced Management Techniques for 
Improving Enlisted Personnel Planning (Advancement, Strength, 


and Training Planning Model) 


2) New Techniques to Forecast Career Input Requirements 
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07 MAN-MACHINE SYSTEMS R&D 


This area is primarily concerned with the implementation 
of human operator considerations in the development, 
operations, and maintenance of new and current organizational, 
weapons and support:systems. Goals include (a) an under- 
standing of, and techniques to quantify and extrapolate, 
meaningful relationships among human performance, equipment 
characteristics, and environmental conditions; (b) development 
and application of man-machine standards, specifications, and 
tradeoff analysis methodologies in systems development, 
planning, and procurement; and (c) techniques for application 
of human capabilities to anticipated demand of future weapon 
systems. 


Typical Projects 


1) Advanced Methods for Enhancing Human Performance in 
Critical Operations 


2) Simulation Technology for New System Personnel and 
Training Requirements Determination and Man/Machine 
Tradeoffs 


08 SELECTION/CLASSIFICATION 


The program area involves development, test, and eval- 
uation of new and improved methods for (a) more accurate and 
effective selection and classification of individuals; 

(b) determining their training requirements; and (c) matching 
the man to the job. Selection usually refers to the process 
of determining which individual or individuals should be 
accepted or rejected with regard to a specific program. 
Classification refers to the process of determining and 
categorizing the appropriate skill qualifications of an 
individual .in such a way that each school or job quota is 
filled with particular individuals most likely to perform: 
successfully. 


Typical Projects 


1) Enhancement of Personnel Effectiveness in Overseas 
Environments (Homeporting) 


2) Enlisted Classification Test Development 
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09 APPLIED SYSTEMS DEVELOPMENT 


Identification of personnel and training requirements, 
establishment of initial operator performance standards and 
maintenance skill requirements, assistance in man-machine 
tradeoff studies, and conduct of studies to enhance man- 
machine compatibility for candidate and prototype systems 
under development. The effort frequently involves development 
of training objectives, review of course and lesson plans, 
and the evaluation of operator and maintenance personnel 
training. 


Typical Projects 
1) ASW Ship Towed Torpedo Countermeasures System (NIXIE) 


2) Integrated Shipboard Communications System (ISC) 


10 CAREER STRUCTURES 


Development, test, and evaluation of new and improved 
techniques for (a) arriving at optimal career progression 
patterns; (b) methods for utilizing occupational, manpower/ 
personnel cost, and other manpower factors data in deter- 
mining optimal combination of grade and skill profiles 
required to satisfy Navy operational requirements; and (c) 
techniques for providing balance between needs of the service 
based on technological projections, and needs of the 
individual related to opportunities for increased responsi- 
bility, advancement, prestige, and compensation. Typical 
outputs are optimum length of careers, optimum level of 
specialization, and optimum amount of fleet experience 
required of each grade of occupational specialty to meet 
changing military requirements. 


Typical Projects 
1) New Electronics Career Concepts 


2) Development and Test of New/Revised Components of 
Officer Personnel System 
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11 PERSONNEL RETENTION 


This is a multi-faceted program exploring and developing 
methods: and techniques for inducing qualified and needed 
personnel to remain on active duty, and programs which meet 
fluctuating needs of the naval service. Included are 
(a) investigations in psychometric technology to predict 
retention for use in selection and assignment; (b) develop- 
ment of more fair and effective evaluation of performance 
through computer-aided analysis and display of performance 
evaluation; and (c) development of improved techniques for 
selection and training of personnel in ratings which 
administer personal services to other naval personnel. 


Typical Projects 


1) Development of Psychometric Instruments for 
Identifying Career Motivated Applicants 


2) Factors Affecting Retention (Personnel Reactions to 
Incentives, Naval Conditions, and Experiences) 


LZ ATTITUDE/MOTIVATION 


Develop sociometric and psychometric instruments and 
conduct controlled experiments for purposes of studying man's 
values, needs, aspirations, prejudices and expectations to 
achieve a better understanding on how they affect his thinking, 
feelings and reactions to people, groups, social issues or 
events in his environment. The objective in achieving such 
an understanding is better prediction and control of an 
individual's behavior, motivation, and performance. 


Typical Projects 
1) Development of Attitudinal Assessment Techniques 


2) Assessment of Attitudes and Motivations of Marginal 
Personnel : 
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13. MINORITY DISCRIMINATION MEASUREMENT AND REDUCTION 


This area is comprised of R&D support for BuPers and Navy 
objectives to intensify and expand efforts to eliminate 
racial, sex, and cultural discrimination and bias in the Navy. 


Typical Projects 


1) Development and Evaluation of Culturally Unbiased 
Instruments and Procedures for Selection of Officers and 
Enlisted Personnel 


2) Analysis of Minority Discrimination in Procurement, 
Discipline, and Advancement 


14 ORGANIZATIONAL EFFECTIVENESS 


Development and evaluation of techniques for measuring 
overall military effectiveness of organizational units under 
various contingencies. This involves the development of 
empirically based criteria and the relative importance of 
criteria which should be applied in making judgments of 
organizational effectiveness. It includes studies of indi- 
vidual and organizational behavior and evaluative research 
into managerial practices and organizational characteristics. 
The principal objective is the enhancement of organizational 
effectiveness in relation to organizational goals and total 
system effectiveness. 


Typical Project 


1) Development of Concepts and Criteria for Organizational 
Effectiveness 


ade DISTRIBUTION/ASSIGNMENT 


This area encompasses research and development in the 
managerial functions of personnel distribution, assignment, 
and rotation as carried out by the Bureau of Naval Personnel. 
The program is concerned with the development and evaluation 
of new and improved techniques and procedures based on 
modern-computer technology for (1) the equitable, quantitative 
and qualitative allocation ("distribution") of the personnel 
and inventory among the ships, aircraft squadrons, and shore 
stations of the Navy; (2) the optimal match ("assignment") 
of specific individuals to particular billets in terms of 
prescribed criteria; and (3) the planned, periodic reassign- 
ment ("rotation") of career personnel to the various 
categories of sea, shore, and overseas duty. Objectives are 
the optimum matching of personnel and billets, improved 
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manpower utilization systems, and the optimum matching of 
Manpower requirements and resources over time. 


Typical Projects 


1) Centralized Enlisted Personnel Distribution and 
Management 


2) New Concepts for Centralized Enlisted Rating Control 


16 R&D RESOURCES MANAGEMENT 


This area includes three subareas dealing with efficient 
use of R&D resources. One deals with the selection and 
evaluation of scientific/research personnel employed by the 
laboratories. Research in this area is aimed at the develop- 
ment of performance evaluation criteria in the areas of 
quantitative and qualitative work output and scientific/ 
-technical contributions. Another subarea concerns the iden- 
tification of significant problem areas, assignment of 
priorities, and selection of tasks to be performed to assure 
maximum return for the "R&D dollar". A third subarea is the 
continuing study of the implementation of research recommen- 
dations by consumers as a basis for future research planning 
and to facilitate the implementation of significant R&D 
products. 


Typical Prej eat 
1) Methodology for Developing and Evaluating GOR-43 


17 RECRUITMENT/PROCUREMENT 


Development, test, and evaluation of methods and techniques 
for (1) attracting-and obtaining required skill composition, 
racial mix, and educational levels and for (2) improving 
selection policies and programs relating to manpower require- 
ments, implications of an all-volunteer force, manpower 
ceilings reductions, and changes in strategic guidance. The 
principal objective is to obtain personnel with the appro- 
priate innate and acquired characteristics to meet all of the 
performance requirements and changing requirements of future 
naval forces. 


Typical Projects 
1) Recruiter Selection Techniques 


2) Evaluation of Enlisted to Officer Transition Programs 
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18 PERSONNEL PERFORMANCE 


Development of on-the-job performance standards and 
criteria and improved proficiency measurement techniques to 
assure that selection, assignment, and promotion factors 
are job-relevant. The goal is maximum readiness of individuals 
and units. 


Typical Projects 


1) Criterion Development for Selection and Performance 
Assessment 


2) Improving Enlisted Performance Evaluation and 
Advancement Systems 
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